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Abstract 

Scholars, practice professionals, and policymakers should welcome the new era of evidence-based program- 
ming and policies, but these constituencies need to be realistic about the complexities, uncertainties, and limitations 
that lie beneath what could easily become a simplistic process. This paper discusses some of the requirements for 
the replication of evidence-based programs, suggesting that many of these underlying assumptions are often not met. 
One of these requirements is the evidence itself, and alternative evidentiary criteria are discussed. Amain theme is 
that even if a well- documented program exists, implementing it in communities on a broader scale requires different 
processes that are less well studied. For example, some alternative approaches to summarizing actionable knowledge 
are offered, including characteristics of effective programs, consensus groups, and the Pathways Nfepping Initiative. In 
addition, strategies are discussed that hold the promise of bringing scholars and community stakeholders together in 
a collaborative process that will build community capacity and create and implement effective programs and services 
on a broader scale. Finally, the research enterprise itself needs to be transformed to more effectively contribute to 
program and system community change. Recommendations for improving the process are offered. 
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The purpose of using developmental research to guide policy- 
making is to make sure that what we do for children and families is as 
effective as possible. We sometimes forget, however, that this job does 
not end with policymaking. Implementation of policy is equally impor- 
tant. No Child Left Behind (NCLB) is a good example. It is a reasonable 
piece of legislation. However, its implementation identified a host of 
problems not the least of which was the tremendous state variability 
in implementation (See, e. g., Porter & Polikoff, Social Policy Report, 
2007). For this reason, those of us interested in the uses of research 
have to attend to policy implementation as well as policymaking. This 
is the main reason that SRCD has both Congressional Policy Fellows, 
who deal with policymaking, and Executive Branch Fellows, who focus 
on implementation. 

Another important distinction is between policy and practice. 
Our tendency is to attend to policy, but practice is egually important; 
and the two reguire different approaches. This issue of Social Policy 
Report (SPR) by Robert McCall addresses the use of research to inform 
practice. Evidence-based practice is a popular idea today. However, as 
McCall points out, we have to be clear and careful about our definition 
of evidence. Some would argue, for example, that only results from 
Randomized Control Trials (RCT's) should be used to support practice 
or inform policy. McCall has an earlier SPR (2004) in which he addresses 
whether RCT's should be the gold standard for evidence in the definition 
of evidence-based. The point is that using the term "evidence-based" 
does not necessarily tell us which practices should be disseminated and 
replicated. In this article McCall also raises the equally important, but 
often overlooked, issue of practice-based evidence. One mistake that re- 
searchers have made, which has limited their influence and effectiveness 
in practical settings, is to assume that communication is unidirectional, 
with researchers being the experts with all the information to offer. In 
fact, policymakers and practitioners have just as much information to 
offer academia as we have to offer them. This is perhaps more true of 
the practitioner community than of policymakers because of the great 
diversity of approaches to practice. 

We are also pleased to have three outstanding commentaries 
by Karen Blase and colleagues, J ean Grossman, and our own hfery Ann 
M:Cabe. All three as usual make important complementary points to 
the main article. We are especially pleased to have the statement from 
Dr. McCabe; she has worked in practice her full professional career and 
has worked hard in several ways to put it on SRCD's agenda. Now after 6 
six years of directing SRCD's Office of Policy and Communications (OPC) 
she is moving on to a new phase in her career. Hence, it is fitting that 
she is able to address one of the issues about which she is passionate 
in this, her last, issue of SPR 

This is a somewhat unusual SPR for us in that it commands a 
different audience than we typically reach with SPR J ack Shonkoff has 
an article in Child Development (2000) in which he discusses the differ- 
ent cultures of research, policy, and practice. This SPR continues that 
tradition. Brooke and I hope that with this issue, our next to last issue 
as editors, we increase our attention to practice, in its multifaceted 
nature, as well as policy. 

Lonnie Sherrod, Ph.D., Editor 
SRCD Executive Director 
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Evidence-Based Programming in the Context 
of Practice and Policy 

Robert B. McCall, Ph. D. 

University of Pittsburgh Office of Child Development 

Toward More Realistic Evidence-Based Program- 
ming and Policies 1 

This is the era of evidence-based programming and 
policies, in which funders and policymakers are demanding 
that the services they fund and the policies they create 
are grounded in empirical research and evidence of their 
effectiveness. Such programs and policies, it is reasoned, 
hold the promise of providing needy citizens with more 
effective services that will contribute to more productive 
and satisfying lives. 

The new value for research and evaluation represents 
a long-sought opportunity for scholars, not only to be heard 
but to influence practice and policy to an unprecedented 
degree. After decades of realizing that research had ex- 
tremely modest influences on policymaking (e.g., Weiss, 
1977, 1988; Chelimsky, 1991; Leviton & Borouch, 1983), sud- 
denly it would appear that research has not only "arrived" 
but occasionally it has become the coin of the practice and 
policy realms. 

But every new opportunity has its risks. For example, 
are scholars up to the task? Do researchers have sufficient 
evidence on service programs and policies? Will scholars 
agree on what is "evidence, " or will they get bogged down 
nitpicking the limits of existing research? Will applied 
professionals glibly declare each program to be "evidence- 
based" if there is some research, any research, justifying 
directly or indirectly some element of the program?Are ser- 
vice professionals prepared to create, implement, monitor, 
and evaluate evidence-based programs? Will policymakers 
demand that scholars simply declare programs "proven" 
or "not proven"; will the "proven" programs be faithfully 
implemented; and will these transplanted programs actu- 
ally work? Will anyone pay to find out if they work in the 
new context, or will policymakers assume they are already 
"proven" to be effective and do not need further evalu- 
ation? The quality of implementation of a program is as 
important to achieving desired outcomes in society as the 
original demonstration of the program, but we do not have 
a well- developed science of implementation. In the long 
run, what do scholars, practice professionals, policymak- 
ers, and communities need to do to cultivate these seeds 
of change and bring them to full flower? 

This paper supports the new emphasis on evidence 
as a basis of practice and policy, but all constituencies— 
research and evaluation, practice, and policymaking com- 
munities^ieed to be realistic about this approach. Thus, 
while this paper emphasizes the complexities, uncertainties, 
and limitations that lie beneath what can easily become a 
simplistic process, it also suggests approaches to deal with 
these issues and still produce usable guidelines on which 
practitioners and policymakers can act. 

IVbre specifically, this paper considers issues in how the 
process of creating and implementing evidence-based social 
and behavioral programming and policies, primarily for chil- 



dren, youth, and families, might proceed. Mich of what is 
presented below derives from the author's and colleagues' 
at the University of Pittsburgh Office of Child Development 
experience with policies, practices, and program evalua- 
tion in one locale (e.g., Groark & MCall, 2005, in press; 
McCall, Green, Strauss, & Groark, 1997; McCall, Groark, 
& Nelkin, 2004), and many proposals discussed here have 
not been studied or evaluated. The presentation accepts 
the existing research literature as a starting point of this 
process, so it focuses on how that literature is summarized, 
communicated, used, and implemented. A maj or point is 
that documenting with rigorous demonstration programs 
and evaluations that a given service program can be effec- 
tive at achieving its goals is only the first step along the 
science-to-practice continuum; what happens after that is 
just as crucial to the success of bringing that program to 
scale and achieving effectiveness in other communities, and 
the evidence for the effectiveness of these implementation 
procedures is quite sparse. 

The programming to be discussed tends to be com- 
munity-based, publicly available services that are nearly 
universal within a geographical area or institution (e.g., 
school) or with broad eligibility criteria (often income), 
such as Head Start, drug and alcohol prevention programs, 
and family support services, rather than traditional psy- 
chotherapy, for example (see APA Presidential Task Force 
on Evidence- Based Practice, 2006; Weisz, Jensen- Doss, & 
Hawley, 2006). The policies most frequently refer to those 
that define, fund, and implement such services. For the most 
part, local (i.e., state, county, city) policies are emphasized 
in this paper, because most services are selected, funded, 
and implemented at these levels, and the evidence for and 
processes to implement local services can be different than 
for national programs (i.e., Head Start). 

The purpose of this paper is to consider and assess 
how this process is often currently carried out, alterna- 
tives to the way evidence is summarized and brought to 
the process, factors in how communities attempt to use 
such knowledge to implement or create services, and how 
the general research enterprise could be broadened to be 
more helpful. The paper argues that the process could be 
improved if several of the procedures suggested were used 
and studied to a greater extent. 

A Brief Personal Interpreted History 

Policymakers and funders have long desired "evi- 
dence" that service programs and policies achieve their 
intended goals, a value previously labeled "accountabil- 
ity". President Iyndon Johnson's Great Society was called 
the "experimenting society" by Campbell (1969), because 
many of the new programs and policies were experimen- 
tally evaluated, even with random assignment or quasi- 
experimental procedures. But such evaluations were often 
very large and expensive and aimed primarily at deciding 
simply whether the program "worked" or "did not work" 
and sometimes whether it was "cost-effective". Unfortu- 
nately, the conclusion often was that the program "did not 
work", and the result, in combination with a conservative 
political climate, was that social program research and 
evaluation temporarily fell into disfavor with policymakers 
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(e.g., Rossi & Freeman, 1993). Perhaps this outcome was 
a case of shooting the evaluation messenger along with, 
or instead of, the program, but it also likely derived from 
the fact that the information provided by such evaluations 
consisted largely of an effectiveness grade for a specific 
program and little else. The lessons learned from the 
"experimenting society" should be kept in mind during 
the current incarnation of evidence-based practices and 
policies. 

Although large-scale social experimentation and 
evaluation passed out of favor, policymakers nevertheless 
continued to want to know whether their policies and 
services were accomplishing their intended purposes, 
and the need for accountability increased as problems 
appeared to increase while resources to deal with them 
diminished. In the author's experi- 
ence, two decades ago at the local 
level, "evaluation" of services 
often consisted simply of counts 
of the number of people served, 
their demographic characteristics 
and geographic locations, as well 
as the specific services they re- 
ceived; later policymakers began 
to demand more information on 
outcome effectiveness— did the 
participants benefit from the service or policy? For ex- 
ample, the National United Way urged its local affiliates 
to reguire grantees to provide some evidence of program 
effectiveness, and the Reinventing Government movement 
urged government agencies to demand accountability, not 
only from its grantees but also from themselves. 

But as millions if not billions of dollars were being 
spent on human services and policies, policymakers appar- 
ently grew impatient with the accountability process. They 
often reguired each new program to conduct an evalua- 
tion, and the program and evaluation would be funded 
for only a few years and enough time for one cohort of 
participants. Moreover, the emphasis often cut straight to 
assessing outcome, skipping the monitoring and evaluation 
of the implementation of the service program and the 
fidelity of the service to its theoretical guiding principles 
(e.g., Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). 
These themes persisted despite occasional demonstrations 
(when rare funding permitted) that service programs often 
needed two or three cohorts of participants for service 
professionals to learn and refine the implementation of 
the program before participants benefited (Fixsen et al., 
2005)— a validation of Campbell's (1987) admonition "to 
evaluate no program before it's proud." This represents 
another lesson learned. 

Finally, policymakers wondered why they should 
try out so many new service programs and wait several 
years to find out if they worked when they could insist on 
replicating locally those service programs that already had 
been tried and found to be effective elsewhere. The strat- 
egy of replicating proven programs became the guiding 
principle of "evidence-based programming and policies", 
and today it is often the primary strategy of local funders 
and policymakers. On the surface, this approach is logical 



and reasonable; beneath the surface it is more complicated, 
uncertain, and potentially limited than it would appear. 

Requirements for Replication of Evidence-Based 
Programs 

In its simplest and most direct form, evidence- 
based practices often consist of identifying service programs 
that have been evaluated and found effective in one loca- 
tion and replicating those service programs in another. An 
example is the nurse-practitioner home visiting program of 
Olds and Kitzman (1993) aimed at high-risk mothers to mini- 
mize abuse and neglect and maximize development of their 
infants and toddlers, which has been widely implemented 
across the country and which has a relatively larger research 
base than many programs. But to replicate proven programs 
as a common strategy requires that 
certain circumstances are pres- 
ent, and these requirements must 
be recognized and are often not 
fulfilled (e.g., Flay et al., 2005; 
Groark & McCall, 2005, in press; 
McCall et al., 2004). 

Requirements for Replicat- 
ing Proven Programs 

1. Some programs must ex- 
ist that have been evaluated and demonstrated to be 
effective. The most obvious requirement is that programs 
must already have been created, evaluated, and found to 
effectively prevent or treat the target problem or benefit 
the participants in the intended way. Some social problems 
have plagued society for years, and the government has 
funded a variety of attempts to prevent or rehabilitate 
such individuals. Teenage problem behavior (e.g., school 
failure, risky sexual activity, substance abuse, delinquency, 
and violence) is a domain in which numerous programs have 
been attempted and evaluated over the years, and this 
research literature, more than many, is up to the task of 
evidence-based programming (e.g., Weissberg & Kumpfer, 
2003; Weisz, Jensen- Doss, & Hawley, 2006). 

But this is not the case in every domain that gov- 
ernment serves, and the reasons for this deficiency vary. 
For example, at least four decades of research show the 
potential of quality early childhood care and education 
programs to improve school readiness, reduce school 
failure and special educational services, improve employ- 
ability, and lessen antisocial behavior and criminality (e. g. , 
Haskins, 1989; M:Call, Larsen, & Ingram, 2003; Ramey & 
Ramey, 1992; Reynolds, Wang, & Walberg, 2003). But it is 
only recently that research is being focused on the effec- 
tiveness of various early childhood curricula that promote 
emerging literacy, numeracy, appropriate social- emotional 
development, and pro- social behavior. Further, there is now 
some evidence on the essential elements of Head Start and 
Early Head Start programming that actually produces the 
benefits that the research demonstrates is possible, but 
this literature is less developed with respect to local early 
care and education services more generally. Here is a case 
in which research preceded policy in describing the poten- 
tial of a human service but for decades was less prepared 
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to substantiate the effectiveness of specific curricula or 
program characteristics. 

Conversely, social problems and policies frequently 
precede research. For example, when youth gangs emerged 
as a social issue in the early 1990's, there were few "proven 
programs" to keep youth out of gangs at the time policy- 
makers needed to act. 

In some other domains we may never have sufficient 
evidence substantiating effective programs, especially for 
preventing low-frequency problems and problems that are 
difficult to solve with circumscribed interventions. Despite 
the great costs to victims and society of child abuse and 
mass adolescent killings in schools, it is difficult to conduct 
the necessary research to prevent such low-frequency 
phenomena in which risk factors are not sufficiently pre- 
cise to identify potential problem 
individuals to study. 

Finally, sometimes the 
government itself has failed to 
invest in evaluation, even for some 
of its largest programs. For many 
years, Congress voted to increase 
the number of children who could 
participate in Head Start, now the 
nation's largest direct service to 
young children, but it did not fund 
evaluations or other research on the program's effective- 
ness. When the program came up for reauthorization nearly 
a decade ago, there was little information on its effective- 
ness (this has now changed; see Ludwig & Phillips, 2007; 
Puma, Bell, Cook, Heid, Lopez, Zill, et ah, 2005). 

Therefore, for a variety of reasons, the research and 
evaluation literature may not be able to deliver "proven" 
programs to policymakers to replicate for every, or even 
most, human service domains. 

2. A program needs to be packaged and described in 
detail sufficient for others to faithfully replicate it. Educa- 
tional curricula, for example, are articulated in great detail 
and packaged in a format and with a variety of supporting 
materials that allow schools and teachers to implement 
them easily and faithfully. This is likely the case because 
schools want packaged curricula and for-profit publishers 
compete to sell such curricula on the basis of their com- 
prehensiveness, quality, and ease of implementation. But 
most other human services have not expected, planned for, 
or had funding to write down and package for dissemina- 
tion a detailed description of the service program, training 
materials, implementation plan, and so forth. Even the 
research literature is lacking full descriptions of service 
programs, despite the facts that the research community 
values descriptive detail and that many of the programs 
evaluated were "demonstration" programs that presumably 
would be replicated if found successful. Consequently, in 
many cases the research literature is in the embarrassing 
position of having demonstrated that a certain "program" 
or "intervention" produced successful outcomes but having 
limited information describing the nature of that "program" 
or "intervention" for either research purposes or program 
replication (Fixsen et al. 2005). 

3. Local service providers must be willing and able 



to faithfully replicate the documented program. Even if 
the program is described completely, local service provid- 
ers may not replicate it faithfully (IVfcCall & B. L. Green, 
2004). Even multi-sited national interventions, such as 
the Comprehensive Child Development Program, which 
prescribed the service regimen in great detail so that a 
uniform intervention could be implemented in numerous 
sites around the country, found that the actual program 
that was implemented differed substantially in character 
from one site to the next (Gilliam, Ripple, Zigler, & Leiter, 
2000). Through training and experience, service providers 
develop their own beliefs and standards about services that 
work, and they are taught to tailor services to the individual 
needs and circumstances of their participants and service 
environment. Some program features may need to be modi- 
fied to conform to local regulations, 
the local standards of best practice, 
available resources, and even the 
dictates of the agency's board of 
directors (Fixsen et al., 2005). 

This propensity is based on 
sound therapeutic rationale, but 
it threatens the logic of replicat- 
ing a program, since the program 
actually implemented may not, 
perhaps should not, be the clone 
of the program demonstrated to be effective. Further, the 
adjustments made to fit the local participants, environ- 
ment, and service providers' skills and beliefs are rarely 
studied, so the generalizability of a service program is often 
unknown, because program replications are rarely studied 
(or published) and policymakers may make the inference for 
economic reasons that a replicated program does not need 
to be evaluated because it is based on a program already 
"proven" to be effective. 

Therefore, service professionals are unlikely to 
faithfully replicate a program, the changes they make may 
or may not be effective but this is not usually known, and 
funders and policymakers need to be encouraged to fund 
both monitoring program implementation to assess the 
fidelity of replication as well as evaluating the program's 
outcomes to assess whether the replication is as effective 
as the prototype. 

4. Replicating research-documented programs as- 
sumes they will produce benefits comparable to the origi- 
nal demonstration project. Even if the original program is 
written down in detail and faithfully replicated, there is 
no guarantee that it will be as effective as its predecessor 
(e.g., L. W. Green, 2001). Indeed, it is widely understood in 
the human service field that "service programs do not travel 
well", that is, they are not as effective when replicated as 
the original demonstration (Fixsen et al., 2005). 

There may be several reasons for this (e.g., Fixsen 
et al., 2005). First, as described above, the program may 
not be faithfully replicated and the adjustments to local 
circumstances may inadvertently limit (or increase) the 
program's effectiveness. Second, not all programs work for 
all participants, and a program that works for one type of 
participant in one part of the country may not be effective 
for other participants in other locations. Third, not all ele- 
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merits of the original demonstration that are crucial to its 
success may be replicated. For example, the commitment, 
enthusiasm, and skills of the creators of the original dem- 
onstration program as well as the particular provider- client 
interactions and relationships that they fostered may have 
been crucial to its success, but those personnel character- 
istics may not have been duplicated by the providers of the 
replication. We commonly recognize that the individual 
characteristics of a teacher contribute substantially to the 
effectiveness of a given curriculum, and this principle may 
be equally applicable to other human services. 

The Evidence Needed to Declare a Program "Evi- 
dence-Based" 

In the philosophy of science, nothing is ever "prov- 
en" with total certainty, because 
cause and effect is probabilistic, 
and in the behavioral sciences those 
probabilities are often modest. But 
policymakers tend to want to know 
whether a given service program 
works— "yes" or "no", not "maybe", 

"sometimes", or "to some extent 
for some people". How much evi- 
dence and what kind of evidence is 
needed to declare that a service 
program is worthy of being repli- 
cated on a larger scale? 

Criteria forjudging the evidence. There have been 
several attempts to set forth criteria for what constitutes 
evidence that a given program "works". At the extremes, 
one set (e.g., Biglan, Mrazek, Camine, &Flay, 2003) simply 
consisted of seven levels of the quality of evidence, another 
(Shaxson, 2005) listed five components of "robustness" 
(i.e., credibility, generalizability, reliability, objectivity, 
rootedness), and another (Flay et al., 2005) presented 47 
standards for efficacy, effectiveness, and dissemination 
of programs. Between these in level of detail, Groark and 
McCall (2008) offered standards of sound program evalu- 
ation that could be used to judge the persuasiveness and 
comprehensiveness of the research literature on a service 
program. These schemes are very general, but others have 
provided guidelines for evidence pertinent to programs 
aimed at specific problems (e.g., Barkham et al., 2001; El- 
liott & Mhalic, 2004) . Unfortunately, these different criteria 
do not always assign the same overall level of quality to a 
given program (Elliott & Mhalic, 2004). 

Crucial elements of judging a literature on a service 
program include 1) the persuasiveness of the research that 
the program can produce the intended benefits under ideal 
and controlled conditions (i.e., "efficacy"), which often de- 
pends on the technical details of the research methodology; 
2) the extent to which the program does produce benefits 
when implemented in the field under conditions and with 
target participant groups, staff, and budgets that are similar 
to those likely to exist when the program is replicated in a 
community (i.e., "effectiveness"); 3) estimation of effect 
size (i.e., odds ratios and cost-benefit analyses; see Scott, 
Mason, & Chapman, 1999) that demonstrate that the pro- 
gram provides sufficient benefits to enough of the people 



participating to justify its cost; 4) reasonable judgments 
identifying (if not evidence supporting) the crucial elements 
of the program necessary for its effectiveness to guide which 
aspects are essential and which are more amenable to tailor- 
ing to local circumstances, and 5) feasibility (i.e., detailed 
descriptions of crucial services and procedures, personnel, 
and budgets necessary to implement the program). 

Reaching consensus. J udging the persuasiveness of a 
research literature requires technical skill, even when one 
or more of the sets of criteria cited above are employed. 
Practitioners and policymakers who do not have training in 
research methodology might seek the advice of specialists; 
and those specialists should recognize that experts do not 
always agree with each other (Elliott & Mhalic, 2004), prac- 
titioners and policymakers are not interested in the level 
of detail that they are, and that the 
desired outcome is a recommenda- 
tion of whether the evidence justi- 
fies replication under the practice, 
policy, and budget circumstances 
that exist in that locality. 

Sometimes this review of 
evidence has already been con- 
ducted and published. For example, 
the Campbell Collaboration (C2) is 
a network of researchers, practitio- 
ners, policymakers, and consumers 
that produces reviews of the effectiveness of interventions 
on targeted groups (Cooper, 1998), and the Centers for 
Disease Control's (2006) Guide to Community Preventive 
Services reviews evidence on a variety of physical and 
behavioral health programs and makes recommendations. 
More specific to children, the What Works Clearinghouse 
(www.w-w-c.org) and the Division for Research on Ex- 
ceptional Children (Odem, Brantlinger, Gersten, Homer, 
Thompson, & Harris, 2005) provide such reviews among 
others (see Winton, 2006, Appendix). 

A similar strategy is the "consensus conference or 
committee, " often conducted under the auspices of the 
National Institutes of Health, the National Academy of 
Sciences, or some other neutral body, in which academic 
authorities on various aspects of an issue are asked to re- 
view the literature in their specific domains and then come 
together to forge a conclusion that represents the best 
consensus on the issue at hand. These conferences range 
from having a highly specific focus (e.g., do immunizations 
containing small amounts of mercury produce childhood 
autism?) to much broader concerns (e.g., what is the po- 
tential contribution of quality early childhood care and 
education? Bowman, Donovan, & Burns, 2001; Shonkoff & 
Phillips, 2000). 

Bringing Evidence to Practice and Policy 

The so-called "consensus" resulting from these pro- 
cesses is not likely to be a simple "proven" or "not proven" . 
The policy maker often needs to decide immediately, can- 
not wait for more research to be done, and must act on 
the "best available research" at the moment even if it is 
methodologically limited, fragmentary, and more uncertain 
than conclusive. IVfiking recommendations for action on the 
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basis of limited and uncertain evidence is not a common or 
comfortable task for most scholars, and the process may 
need non-research professionals and others to fill in the 
knowledge gaps and craft recommendations that fit prac- 
tice, social, fiscal, and political circumstances, standards, 
and constraints. In short, evidence-based programming and 
policymaking may need to be more complex and compre- 
hensive than simply replicating "proven programs", and the 
processes involved in implementing evidence in local com- 
munity programming are just as crucial to providing service 
benefits as having a "proven program" but with much less 
evidence to support them. 

Those who wish to go beyond simple service program 
replication emphasize three themes, which are not mutually 
exclusive. The first represents more comprehensive and 
flexible methods of reviewing the information available and 
casting it in a form that is more readily usable in creative 
and flexible ways by practitioners and policymakers. The 
second theme emphasizes new approaches to translating 
research into practice and creating and implementing 
evidence-based services in communities. The third theme 
proposes broadening the research enterprise itself to be 
more directly relevant to the information needs of those 
who desire to create broad systems of services and policies 
for children and families. 

Alternative Approaches to Summarizing Actionable 
Knowledge 

The first theme is represented by two strategies 
that attempt to broaden and improve the utility of the 
information brought from existing research and practice 
beyond the effectiveness of specific, potentially replicable, 
service programs. These suggestions are based on the prem- 
ise that simply replicating service programs one by one does 
not permit much flexibility, adaptability, or innovation nor 
does it create a uniform system of services in a particular 
locality. 

Characteristics of effective programs. Rather than 
focus solely on delivering one or more packaged service 
programs for replication, scholars and practitioners also 
should provide policymakers with characteristics of suc- 
cessful programs. Some broad-based consensus groups have 
done this with varying levels of deliberateness and speci- 
ficity. Specific characteristics would provide guidelines to 
practitioners on what elements should compose a service 
program, and policymakers can evaluate the merits of ser- 
vice program proposals for funding against these criteria 
(e.g., Groark & McCall, 2005; McCall et ah, 2004; Nation et 
al., 2003). Particularly useful, but not commonly available, 
is a strong theory of change and evidence of how or why a 
program works that could guide replications and variations 
of a service. 

Characteristics of successful programs might be of 
two kinds, general and specific. When characteristics of 
successful programs have been listed for human services 
pertaining to a variety of different prevention and treat- 
ment goals and domains, many of the same characteristics 
appear on each list. For example, Table 1 presents one 
integration of several lists of characteristics of successful 
programs in early childhood care and education (M:Call, 



Larson, & Ingram, 2003), family support (Layzar, Bernstein, 
& Price, 2001; Schorr, 2003) and adolescent problem behav- 
ior prevention services (Kumpfer &Alvarado, 2003; Nation, 
Crusto, Wandersman, Kumpfer, Seybolt, Morrissey- Kane, & 
Davino, 2003). These general characteristics in turn can be 
supplemented by characteristics that are more specific to 
the particular type of program and the problems or skills 
that are its focus. 

This approach is not intended to replace reviews of 
specific program effectiveness, but to supplement them, 
and it has certain additional advantages. For example, 
characteristics of successful programs permit flexibility, 
adaptation, and creativity at the local level. Further, they 
likely are more stable across changes in evolving research 
literatures than specific program evaluations. 

On the other hand, characteristics of successful 
programs are rarely studied directly with deliberate ex- 
perimental manipulation or non- randomized comparisons; 
even outcomes across studies with versus without a specific 
characteristic are rarely systematically compared. Instead, 
these characteristics are often simply those that are com- 
monly represented in the programs that have been shown 
to be effective. Therefore, they are likely to be confounded 
with other factors, all may not be necessary for program 
effectiveness, and little direct evidence likely will exist to 
document that any single characteristic is necessary or even 
helps produce successful outcomes. But these characteris- 
tics do represent conventional wisdom or "best practices" 
that have been associated with successful outcomes, and 
they potentially represent a partial set of criteria for 
policymakers to use in crafting Reguests for Proposals and 
selecting those to be funded and for practitioners to use 
as a foundation for program development and as standards 
of implementation. 

Communities of practice. Acentral concern of prac- 
titioners is that the research literature often does not tell 
them what really matters in practice (e.g. , Buysee & Wesley, 
2006, Foreward). Conseguently, practitioners sometimes 
form "communities of practice" (Lave & Wenger, 1991), 
which are composed of people who have a common prac- 
tice interest who share their perceptions and observations 
and learn from each other's everyday experience (Wesley 
& Buysee, 2006). 

The members may or may not communicate in- 
formation from the research literature, depending on the 
backgrounds of the people involved. But the simple dissemi- 
nation of research information is not likely to be sufficient 
to prescribe what should be done in practice. This is why 
many definitions of evidence-based practice and policies 
emphasize that it consists of research evidence integrated 
with professional wisdom (e.g., Buysee & Wesley, 2006), 
clinical expertise, and professional and client values 
(e.g., Sackett, Straus, Richardson, Rosenberg, & Haynes, 
2000). Thus, communities of practice should involve both 
practitioners and researchers, and as such they have the 
potential to blend research and practice, adapt evidence- 
based programs to fit specific circumstances, and fill gaps 
in the research literature with "best practices" based on 
professional experience. 

Pathways Mapping Initiative (PMI). The Pathways 
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IVfepping Initiative (PM) of the Project on Effective Interven- 
tions at Harvard University represents an approach devel- 
oped by lizbeth B. Schorr and colleagues (Schorr, 2003) that 
consists of a systematic attempt to broaden the knowledge 
brought to the practice and policy tables beyond traditional 
research evaluations of specific service programs and to 
integrate it with professional experience and values. Pro- 
ponents of PM regard the traditional knowledge currently 
available to communities as inadequate because it comes 
in small, isolated, and disjointed pieces; often arrives too 
late; is derived from a severely limited range of interven- 
tions that have been and can be elegantly evaluated; and 
usually fails to identify what really made the intervention 
work. 

The PM process attempts to broaden the knowledge 
base about "what works" by applying reasonable judg- 
ments and plausible interpretations to a preponderance of 
evidence culled from accumulated professional experience 
as well as program evaluation findings, both coupled with 
strong theory. Further, PM's information is developed, 
organized, and presented in a way that is intended to help 
communities themselves to think coherently and systemati- 
cally across programs and across systemic and disciplinary 
boundaries to determine 1) the combination of actions 



needed to produce a desired outcome, 2) the key ingredients 
that likely make those actions effective, and 3) the com- 
munity contexts that will influence effectiveness. 

The Mental IVbpping Process that PM uses is similar 
to the typical "consensus conference" described above, 
but it extends beyond the research evidence by convening 
groups of highly knowledgeable, experienced, and diverse 
individuals including researchers, practitioners, policymak- 
ers, and other potential stakeholders. The process asks 
these individuals, "considering the evidence from research, 
theory, and experiences you have been exposed to over the 
years, what could a community most effectively do if it were 
committed to achieving the specific outcome under consid- 
eration (e.g., higher rates of school readiness, improved 
family economic success)?' Further, the goal is not simply to 
identify effective programs but to identify the elements of a 
community-wide system, despite limited relevant research, 
that will contribute to improved outcomes. 

Such a process also concentrates on filling the gaps 
in knowledge as well as identifying current interventions, 
services, and supports to forge new connections among 
them. For example, when participants were asked to focus 
on improving school readiness for the highest risk children, 
they did not recommend new programs or replications of 



Table 1. 

Some General Characteristics of Effective Behavioral Child and Family Intervention Programs 

Program Characteristics 

Theory based. Program has a conceptual rationale and evidence that the interventions should 
produce the desired outcomes. 

Family focused. Services are aimed directly at all relevant family members and attempt to 
improve family relationships and dynamics to create a mutually supportive family environment 
and establish social support for family members from the community. 

Appropriate timing. Services are initiated when people need and are receptive to them, 
typically early in the development of the target behavior or problem. 

Sufficient dosage. Services have sufficient duration (i.e., months from start to finish), 
frequency (i.e., service contacts/ hours per month), and density (i.e., percent time engaged in 
activities known to contribute to positive outcomes); higher dosages may be needed for 
participants at greater risk. 

Accessible. Services are readily and conveniently obtained (e.g., transportation, child care 
provided), appointments easily arranged, services promptly delivered. 

Matched to participants. Services are sensitive and matched to the needs, culture, 
developmental level, and circumstances of the participants. 

Personnel Characteristics 

Well educated and trained personnel. Personnel are both generally well-educated and well- 
trained specifically for their responsibilities. 

Good supervision. Front-line personnel are continuously monitored and supervised in a 
thoroughly supportive (perhaps "reflective") fashion by competent administrative personnel. 
Structural support. Personnel are structurally supported by providing them enough time, small 
caseloads or few children, space, equipment, consultants, specialized services, and supportive 
colleagues. 

Relationships. Personnel develop relationships with participants that are warm, trusting, 
empathic, and sincere, and personnel are perceived by participants to be competent, genuine, 
and concerned. 

Participant Characteristics 

Engaged. Participants are engaged in the program, attend and participate in services and 
program activities, persist in the program until graduation, and support family members and 
friends in their participation. 

Social-behavioral risk. Social-behavioral services are most effective with participants at 
greatest social-behavioral and economic risk, rather than at medical risk. 



Based upon reviews of family-based services, early childhood care and education programs, and adolescent 
problem prevention programs (for more details, see Kumpfer & Alvarado, 2003; Layzar, Goodson, Bernstein, & Price, 
2001; MiCall, Larson, & Ingram, 2003; Nation, Crusto, Wandersman, Kumpfer, Seybolt, M)nissey-Kane, & Davino, 2003; 
Schorr, 2003). Reprinted from Groark & NfcCall (2005) with permission. 
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specific programs but emphasized the connections that must 
be built between existing programs and institutions to make 
it easier to bring needed help to disconnected families. 
To guard against being swayed by the enthusiasm of one 
or two individuals or current "fads" in best practices, PM 
takes care to ensure a rich mix of backgrounds and out- 
looks among participants and to develop strong consensus 
across these diverse perspectives. The first PM Pathways 
addressed school readiness, successful transition to young 
adulthood, and the prevention of child abuse and neglect 
(www. PathwaysToOutcomes. org). 

The most distinctive features of the PM process are: 

Bridge disciplines and go beyond the traditional 
evaluation literature to identify the specific actions (i.e., 
services, supports, and other interventions) in many diverse 
arenas that contribute to 
achieving specified outcomes; 

Identify easy-to-use in- 
dicators of progress toward 
specified outcomes; 

Describe the essential 
elements needed for success 
with precision and detail, 
even in the absence of ab- 
solute certainty, to increase 
the likelihood that promising 
efforts can be taken to scale 
or transported successfully to new environments; 

Identify the elements of community contexts and 
system infrastructures that support and sustain effective 
change over time to clarify the limitations of individual 
programs and illuminate synergy in the work ethics across 
domains. 

As yet, the PM Pathways process or products have not 
been empirically evaluated. 

Improving Community Capacity to Create and 
Implement Evidence-Based Practices 

Holding consensus groups and PM workshops that 
summarize the research evidence and characteristics of 
successful programs blended with multi-disciplinary knowl- 
edge of best practices are examples of summarizing "what 
works. " But is this sufficient? Not likely. First, there is a 
dissemination gap— what scholars know to be important 
and effective (or not) is often not adequately communi- 
cated or followed by practice professionals (Ringwalt et 
al., 2002; Wandersman & Florin, 2003). For example, the 
DARE programs to prevent drug abuse had been adopted in 
80%of the elementary schools in the United States despite 
their limited effectiveness (e.g., Enett, Tobler, Ringwalt, & 
Fewling, 1994; General Accounting Office, 2003). Second, 
while dissemination of information to practice professionals 
and policymakers could be improved, it is naive to believe 
evidence of effectiveness alone will lead to successful pro- 
gramming (e.g., Bero, Grilli, Grimshaw, Harvey, Oxman, & 
Thomson, 1998; L. A. Green & Seifert, 2005). Other factors, 
such as local social context, cost, and political issues will 
play a role, and so will the capacity of agencies and even 
the larger community to use the information to create and 
implement new and successful programs. 



Therefore, having in hand a well-documented 
model service program with demonstrated effectiveness 
is certainly helpful, perhaps even necessary, but probably 
not sufficient. The process of getting communities to want 
the program, modifying the demonstrated program to fit 
local circumstances if necessary, and having the program 
implemented with reasonable enthusiasm and fidelity by 
agencies and staff who did not create the original program 
is equally necessary to "bring the program to scale." But 
there is only limited research on the effectiveness of such 
processes (e.g., Fixsen et al., 2005). 

The gap between science and practice resides not 
only in communication failure, but also the inability of 
some communities to implement quality programming and 
make systemic changes. Chinman et al. (2005) identify four 

broad factors that contribute 
to the gap between science 
and practice and that require 
community capacities. First, 
implementing high-quality 
service programs is a complex 
process, requiring a significant 
amount of knowledge and 
skills that are far beyond 
following a service practice 
recipe. Second, systems fac- 
tors pertaining to coordina- 
tion among different agencies and community readiness to 
adopt and maintain new strategies must be accommodated. 
Third, having sufficient financial, technical, and personnel 
resources is a typical challenge. And fourth, local clienteles 
and circumstances may pose unusual difficulties, and there 
may be little guidance from the research literature. 

The typical approaches to building community ca- 
pacity focus on technology transfer (Backer, David, & Soucy, 
1995), which assumes that community capacity is limited 
by a lack of information that can be remedied by training 
and technical assistance. Chinman et al. (2005) reviewed 
the variety of training programs offered in substance abuse 
prevention, for example, and concluded that, while helpful, 
they have serious limitations, often pertaining to the appro- 
priateness of the content and barriers to incorporating the 
information into practice. Similarly, having an intermediate 
set of professionals provide direct, hands-on technical as- 
sistance has a good rationale (Fixsen et al., 2005); but the 
literature (Chinman et al. , 2005) suggests that even no-cost 
technical assistance may not always be welcomed, some 
minimum level of community capacity is required to fully 
utilize technical assistance, and community organizations 
are better able to implement some types of assistance (e.g., 
planning, implementation, organizational maintenance) 
than others (evaluation and data analysis). Further, while 
studies of technical assistance have demonstrated improved 
community processes, few studies have examined whether 
participants have improved outcomes. 

The development of community capacity appears 
to be broader and deeper than these strategies can readily 
accomplish. The need for community capacity building goes 
beyond the ability to replicate evidenced-based programs; 
what is needed is a collaborative community process (L. 



Consensus groups and 
communities of practice fill in the 
gaps in evidence. 
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W. Green, 2001) that will create not just programs but 
integrated systems, rooted in theory and evidence, and 
appropriately nestled in the financial, political, personnel, 
clientele, needs, and assets of a particular community. The 
literature (Chinman et al., 2005) suggests at least four re- 
quirements of such a community capacity building process. 
First, the community must have genuine member involve- 
ment and commitment to the process of improvement. Sec- 
ond, organizations and the wider community must possess 
skills in a variety of domains (e.g., managing an effective 
community organization, planning and implementing qual- 
ity programming, matching program to local circumstances 
and resources). Third, existing and new resources must be 
identified, acquired, and managed. Finally, there must be a 
sense of collective efficacy or power to be able to guide and 
manage the skills and resources toward effective outcomes. 

One approach to building community capacity is to 
involve research, practice, 
funding, policymaking, and 
opinion leaders in a collab- 
orative planning process, 
for example one led by an 
independent moderator that 
blends elements of strategic 
planning, needs assessment, 
logic models, a theory of 
change (e.g., Chen & Rossi, 

1983, 1987), and monitoring 
and evaluation strategies. 

Several outlines of the steps 
(Fixsen et al., 2005) and materials to guide the process 
are available. For example, a Strengths, Weakness, Oppor- 
tunities, Threats (SWOT) analysis (Kearns, 1992) provides 
a structure for general strategic planning. The Kellogg 
Foundation (1998, 2000) offers a handbook for conducting 
a generic logic model, Benson (1997) presents an asset- 
based needs assessment of adolescents, and Catalano and 
Hawkins (1996) describe risk and protective factors and sug- 
gestions for program development for preventing antisocial 
behavior. Further, the Getting to Outcomes (GTO) process 
(Wandersman, Imm, Chinman, & Kaftarian, 1999, 2000) has 
been packaged into a manual published by RAND Corpora- 
tion (Chinman, Imm, & Wandersman, 2004; available at 
http://www.RAND.org/publications/TR/TR101), the Search 
Institute (Fisher, Imm, Chinman, & Wandersman, 2006) has 
blended Getting to Outcomes with adolescent asset and 
needs assessment, and the Getting to Outcomes approach 
is available in an interactive web-based technology system 
called iGTO (http://www.kithost.net/ igto/). 

The GTO, for example, represents a structured logic 
model process that systematically guides an organization 
or community, perhaps using an independent moderator, 
through a planning process designed to build community 
capacity, generate enthusiasm and commitment (i.e., "buy 
in") by community stakeholders, and create and implement 
effective service programs and systems. The GTO process 
includes ten generic steps, although the examples often 
pertain to specific goals (e.g., substance abuse prevention, 
sexual and domestic violence prevention; e.g., Wanders- 
man, 2003): 



1. What are the needs and resources available? 

2. What are the goals, target population, and 
desired outcomes? 

3. How does the program incorporate scientific 
evidence and best practice? 

4. How does the program fit or complement other 
programs already in existence? 

5. What skills and capacities are needed to implement 
a quality program? 

6. What are the elements of program implementation? 

7. How will the quality of program implementation 
be assessed? 

8. How well does the program produce its intended 
outcomes? 

9. How will continuous quality improvement strategies 
be incorporated? 

10. If successful, how will the program be sustained? 

Step 3 is the primary 

juncture for injecting theory 
of change and research evi- 
dence. For example, what 
is the rationale (theory) and 
evidence that one versus an- 
other action (services, legisla- 
tion) is likely to produce the 
desired outcomes, why should 
those services produce those 
benefits, and under what 
circumstances will they be 
successful? 

Athorough planning process, such as SWOT and GTO, 
perhaps using a collaboration of scholars, evaluators, service 
professionals, funders, and policymakers, is a good begin- 
ning, but the same collaborators are also likely necessary 
to structure the implementation and oversee the operation 
of a new service program (Fixsen et al. , 2005; Groark & Mc- 
Call, 2008), and some communities and agencies may need 
assistance in conducting those processes. Although widely 
advocated in some circles to achieve the goals mentioned 
above, little research is available on the effectiveness of 
these approaches (see Fixsen et al., 2005). 

Revising the Research Enterprise and Integrating 
It with the Community 

Capacities should not only be built in the commu- 
nity; the capacity of scholarship to improve the process, 
services, and community systems also needs improvement. 

In many quarters of behavioral science, basic re- 
search rooted in theory and aimed at describing general 
cause-and-effect principles preferably with randomized 
research designs that emphasize internal validity has 
dominated the academic value system; in contrast, applied 
research has sometimes been ignored, if not denigrated 
(Groark & McCall, 2005; McCall, 1996; McCall &B. L. Green, 
2004). Perhaps as a partial result, those seeking to improve 
community services and systems lament that social and 
behavioral sciences have not sufficiently studied many of 
the most crucial questions needed to achieve these broader 
practical outcomes (e.g., Auspos & Kubisch, 2004; Chinman 
et al., 2005; Wandersman & Florin, 2003). These different 



Implementing a program in the field 
often benefits from a program 
development process, 
such as a logic model. 
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voices converge on the principle that to improve services, 
service systems, and policies the research community 
must accommodate to the needs of practice and policy by 
broadening its methodological value system and studying 
more directly topics that are necessary for broad system 
and policy changes. 

The need for a science of practice and implemen- 
tation. While MH's emphasis on "translational research"— 
translating basic research knowledge into practice's a 
worthy pursuit, practice fields need a science that studies 
how service professionals actually practice, how programs 
are successfully implemented, and the characteristics of 
service delivery that contribute to participant benefits. 
Fixsen et al. (2005), in their exhaustive review of imple- 
mentation research, argue that an evidence-based program 
will not produce benefits unless it is properly implemented, 
and we know empirically very little about implementa- 
tion, leaving practice fields 
with "the paradox of non-ev- 
idence-based implementation 
of evidence-based programs" 

(Drake, Gorman, & Turrey, 

2002). The inattention to im- 
plementation, both scientifi- 
cally and professionally, may 
explain why services often do 
not work, why they do not 
"travel well" from originating 
to replicating agencies, and 
why simply replicating proven 
programs may not always be a successful science-to-practice 
strategy. Effective implementation is a complex enterprise, 
needs serious expertise, reguires cumulative knowledge 
over repeated implementations, and often takes time and 
several cohorts of participants to perfect. This is why suc- 
cessful implementation of evidence-based programs may be 
accomplished best by "purveyors"— groups who specialize 
in the implementation of specific service programs who 
collaboratively work with agencies to establish that service 
(Fixsen et al. , 2005) . At the same time, funding agencies and 
scholars need to support and study practice and implemen- 
tation per se to complement research on service programs. 

A science of community change. Some scholars ar- 
gue for an even broader science of community change (e. g. , 
Auspos & Kubisch, 2004; Chinmanet al., 2005; Wandersman, 
2003; Wandersman & Florin, 2003). They cite four major 
reasons for advocating a different paradigm for developing 
and using evidence to influence policy, practice, and com- 
munity change. 

First, the traditional and valued scientific para- 
digms are not likely to be applicable. Because a single 
entire community may be the object of change, sampling 
and the classic randomized control trial are not suitable. 
Anthropology's methods of studying a single culture may be 
more relevant. Further, many focal intermediate and long- 
term outcomes (e.g., community support, collaboration 
among organizations, motivation of clients to participate 
in programs) are not easily guantified, and demonstrating 
improvements in an entire community or organizational 
system represents a much more difficult research task than 



showing a specific measurable benefit in a highly-targeted 
group of 30 individuals. Also, the length of time needed to 
achieve community change is much longer than the patience 
and resources of traditional social and behavioral research 
funders. As a result, community change must be studied with 
different methods, it reguires more time, and it likely will 
provide answers with less certainty than many traditional 
scholars and funders have come to expect. 

Second, community-change initiatives place a 
strong emphasis on community involvement, community 
choice, and community building. These community-building 
activities are not under the researcher's control, must be 
tailored to local circumstances and adapted over time, 
and often do not turn out to be the ideal strategy based 
dispassionately and uncompromisingly on existing research. 
Conseguently, the community (the object of study) must 
be an active participant in the design of its own program 

(the independent variable) 
and evaluation (data collec- 
tion) to motivate participants, 
to match the program to the 
unigue characteristics of that 
community, and to implement 
the program in the context of 
the finances, power, and poli- 
tics of that environment. The 
GTO and similar approaches 
described above are aimed at 
this theme. 

Third, a uniform 
intervention is not likely to be appropriate when so many 
facets of a community's system reguire change. Commu- 
nities are often highly decentralized, contain numerous 
stakeholders and power groups, and have multiple and 
discrete funding streams each with their own complex sets 
of rules and regulations, and policies are influenced or set 
by a diverse array of philanthropists, administrators, and 
legislators at the local, state, and federal levels. And while 
they must work within the influence of these constraints, 
community-building professionals operate largely at the 
ground level and consist of a diverse set of organizations 
and groups that are typically small, unaffiliated, and work 
independently rather than as part of a larger network. 
Therefore, no single mechanism is likely to easily produce 
change throughout this diverse field. Thus, the resulting 
literature will be a collection of case studies from which 
more general themes must be inferred. 

Finally, actual change in communities in specific 
services and in community systems is more likely governed 
by political, ideological, or fiscal priorities than by research 
findings. Such changes occur via many different avenues and 
at the hands of many different people, and the process is a 
continuing one of lessons learned, feedback, revision, and 
evolution conducted over a period of time in which many 
of the maj or players, including political administrations, 
may change. 

Community science. Some scholars (e.g., Chinman 
et al., 2005; L. W. Green, 2001; Wandersman, 2003; Wan- 
dersman & Florin, 2003) recognize the value of the predomi- 
nant evaluation model and rigorous research methodology, 



Current translational research does 
not go far enough toward 
studying implementation and 
community change. 
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Commentary 



A Case for Replicating Proven Programs 
J ean B. Grossman 

McCall addresses a very important issue: "How can society best learn from and utilize the findings from research 
to improve the efficiency of its social programs?" Mich of the paper is devoted to examining how researchers and 
practitioners can work and learn together to foster better alignment between practice and theory. While in one sense 
this can be thought of as "evidence-based" programming, most policymakers or funders use the term differently. When 
they refer to "evidence-based programming," they often mean the replication of proven program models. 

McCall addresses replication in the beginning of his paper. He rightly notes that there are unfortunately few such 
proven programs, and thus argues for more research-based innovation in program practice. The problem is that we 
cannot assume these innovative new practices are effective— they are promising perhaps, but not proven. 

There is definitely a role for social entrepreneurs fostering innovation, but there is also an important role for 
replication. As Racine states, the nonprofit sector "cannot thrive solely on the celebration of differences, especially 
not when communities have problems and challenges in common." 1 There is a widespread belief that that every 
program must be tailored to individual locales because the people and issues are surely different from community 
to community. However, people's taste differ and yet, Starbucks is more or less the same all over the country. The 
tendency to want to change and adapt programs has undermined many "replication" efforts. Decades of evaluation 
research has shown that it is actually very difficult to improve the lives of individuals above and beyond what would 
happen to them without programmatic intervention. Most social programs are built on strong theory, but the devil is 
in their operational detail as to whether they actual improve lives. Given the paucity of truly proven programs, it is 
critical that we find ways to effectively reproduce the ones that work. 

Let me then expand on an area that M:Call only touched on, namely, what it takes to replicate proven programs. 
All too often new sites modify successful models to make them "better fit" their local context, without identifying the 
core elements that made the model effective in the first place. This doesn't mean that replicated programs have to look 
exactly like the "mother" site; in fact, some variation is probably necessary to meet local needs. But it's important to 
ascertain which program elements can be changed and which ones are absolutely vital to producing positive results. 

Programs that want to expand should spend the time and money necessary to thoughtfully and carefully replicate. 
The Edna McConnell Clark Foundation, BridgeSpan and Public/ Private Ventures have been helping organizations that 
have strong evidence of success expand for more than a decade, doing so in a very methodical manner. They have 
learned that replication cannot be done quickly or on the cheap (Racine, 2003, 2004; Roob and Barbach, 2009; Sum- 
merville, 2009). Aprogram's critical elements must be identified. Intentional training, support and communication is 
needed for several years. Funding strong infrastructure (e.g., the hire of a strong director and having a data system), 
while not the direct services most donors want to support, is imperative to successful replication and ultimately the 
delivery of strong services. Finally, continuous monitoring and evaluation is necessary to ensure the program, which 
will and should change over time, continues to run as it is intended. 

Society does need to discover or create new programs in areas where there are no proven ones. However, the 
bias in the nonprofit world is for everything to be home grown and local. Given how hard it is to engineer change in 
the natural courses of people's lives, we should invest more in successfully spreading proven programs than we do now. 
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Commentary 



The Silo Problem 
Mary Ann McCabe 

This Social Policy Report by Robert McCall, who is an expert at bridging research with practice and policy, raises 
important and timely issues. His report summarizes the history and current state of the field of evidence-based practice 
and programming. There is widespread and growing support for using research evidence to support investments and 
to optimize outcomes - across fields as diverse as education, health care, and social services. However, we continue 
to witness obstacles to effective translation, dissemination, diffusion, adoption and implementation. It is important 
to maintain a broad definition of "evidence-based practice", similar to that promoted by the Institute of Nfedicine 
(2001) which includes the integration of the best available research information with practitioner wisdom and experi- 
ence, consumer/ client culture and values, and available resources. It is equally important to note that we continue 
to observe an emphasis of evidence-based practice over practice-based evidence or re search- tested programs that 
attend as much as possible to real world settings and constraints. 

Clearly, sharing evidence about programs proven to be effective in the research setting, or dissemination, is a 
necessary first step. In this regard, McCall discusses considerations forjudging "evidence", or actionable knowledge, 
and methods for arriving at consensus. However, he highlights the additional obstacles to community implementa- 
tion of programs. For example, he identifies the fundamental assumptions underlying replication of re search- based 
programs and practice that are often not met. Among these is identification of the critical components in programs 
that make them effective which are the most vital aspect to replication outside the research setting. Replication is 
challenged by the all too frequent "silos" of academia and the community. 

Communities are complex systems, as are the organizations and people within them. Therefore, the process for 
communities to adopting new programs and implement them with fidelity to the model program in research is neither 
simple nor straightforward. McCall describes approaches to building community capacity in this regard. He also points 
out that the actual implementation of programs, and program outcomes, need continued study in the community. 
Again, this requires ongoing collaboration between researchers and those overseeing program implementation - whether 
practitioners, administrators or policymakers. 

This Social Policy Report emphasizes the importance of overcoming the "silo problem". It highlights the essential 
role of collaboration for evidence-based practice and programming— collaboration across researchers, practitioners, 
intermediaries (also known as knowledge brokers or purveyers), federal, state and local policymakers, funders and 
opinion leaders. McCall proposes innovative methods for enhancing collaboration among these groups to drive greater 
success. 

However, additional "silos" also need to be overcome and therefore warrant mention. On the continuum of 
basic research to applied research to translation for practice, there are multiple opportunities for the failure of dis- 
semination, diffusion, transfer or uptake. And yet there are critical pieces of information in these varied types of 
research for effective practice and community programs. Further, each area of programming for children and families 
that seeks to employ evidence-based practice— early education, education, mental health, health, health promotion, 
substance abuse, child welfare, and so on— does not typically learn from another field about implementing research- 
based programs, or effective knowledge transfer processes. The academic and professional literature, and even social 
networks, in each field are often quite distinct. Similarly, service delivery systems are also typically very separate, so 
that lessons learned about implementing re search- based programs are not likely to be shared— even within the same 
community. Finally, funders very often operate independently of each other in their methods for encouraging transla- 
tion and implementation of programs in communities, such that successes and failures cannot be shared very easily. 

The field of developmental science can and should play a leading role in bringing research evidence to practice 
and policy in order to improve the lives of children and families. We welcome and applaud the interest of policymakers 
and funders in turning to research evidence to inform their investments, and hope they will continue to recognize the 
many types of evidence available to inform different aspects of programs and practice. However, it will be impera- 
tive for them to invest in studies of program implementation and outcomes. Scientists will be wise to increasingly 
collaborate with practitioners, program administrators, and policymakers, in order to conduct research that is well 
informed by real world practice and policies. In addition, they might begin research with an eye on the translation 
and dissemination process, including key partners for bringing research knowledge to communities. This Social Policy 
Report reminds us that if we keep doing what we have always done, we will get what we have always gotten. The 
challenge now is to do even better. 
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Commentary 



"Evidence-Based Programming in the Context of Practice and Policy" 

Karen A. Blase, Melissa Van Dyke, and Dean Fixsen 

"All organizations are designed, intentionally or unwittingly, to achieve precisely the results they get." 

R Spencer Darling 

New paradigms do not fare well in existing systems, and evidence-based service delivery in the real world 
requires a paradigm shift. We appreciate the author's thoughtful articulation of the complexities related to imple- 
menting evidence-based practices and programs in the real world. Research, practice, capacity building, and policy 
are critical to moving evidence-based services from boutique operations for a relative few, to widely accessible pro- 
grams and practices that benefit consumers, communities, and society. An important, overarching theme is that the 
practice and science of implementation must be utilized and improved in order to make better use of intervention 
research results. Afore rigorous research on the "what"— the intervention— will not tell us "how" to implement with 
fidelity and good outcomes over time and across practitioners in complex settings. 

Understanding the contributions and limitations of rigorous intervention research relative to implementation 
is critical. Information about scientific rigor, effect size, and target population can help communities explore whether 
a given program or practice meets a need in their community. Scientific rigor is important. Choosing well is important. 
Implementation is hard work, and we need to be convinced that the effort will be 'worth it' in terms of the desired 
outcomes. However, choosing well will not help the community implement well (Fixsen et ah, 2005). 

Implementation involves installing the infrastructure and processes needed to initiate and sustain effective 
services over time and across practitioners. This means communities need to acquire implementation knowledge 
regarding "how" the practice will come to life in their community. How will organizations need to change to support 
the new way of work? How should practitioners be selected, trained, and coached? How will fidelity be measured and 
outcomes monitored? How will the broader system of services, supports, policies, and funding streams need to change 
to support a new way of work? And, who will do all this work? Some of the considerations and challenges related to 
the "how" include understanding that: 

Changes will need to occur at the practice, organization, and systems levels. Targeting change at the practitioner 
level alone will not result in sustainable practice improvement. 

Policies need to change to enable practices. We need policies to enable practice, and we need practice to inform 
policy (see www.scalingup.org). 

There are significant funding barriers to supporting the implementation of evidence-based practices. Funding 
and support for the infrastructure to create and sustain practitioner competence is a 'forever' cost and is essential 
to producing consistent outcomes across generations of practitioners and managers. If sustainable training, coaching, 
mentoring, and data systems are not in place, then the practitioner competence will not be changed, improved, or 
sustained and benefits will not be realized. 

Communities, agencies and funders will need to be disabused of the notion that we choose evidence-based pro- 
grams and practices because we know they work, and therefore we will not have to fund process (fidelity) and outcome 
evaluation. Evidence-based programs and practice work in the real world because practical program evaluation is used 
to inform decision-making. Eliminating this piece of the infrastructure is a bit like taking a boat for spin, determining 
that the rudder and compass work and then tossing them overboard to save time and money (Biglan & Ogden, 2008). 

Communities need knowledgeable partners to make full, effective, and sustainable use of evidence-based pro- 
grams. Purveyors and intermediaries are critical to developing efficient and effective processes to create the necessary 
conditions for success, and they represent new roles, functions, and structures (Olds, Hill & O'Brien, 2003). While 
scholars may contribute to understanding these emerging entities and functions, it is not likely that they will participate 
in directly providing such implementation services and expertise. Universities simply are not designed to support or 
reward such functions, and the skill sets required do not directly map onto the skill sets of researchers and scholars. 

There is growing recognition that there are important implementation research agendas to be pursued related to 
effective implementation and organizational and systems change. As the author notes, traditional randomized trials 
are not our sole process for creating new knowledge and, in the case of implementation research, are not likely to 
be practical, affordable, nor are they likely to tell us what we need to know about the multitude of complex factors 
that interact across stages of implementation (Panzano, et ah, 2004). We have policy, methodological, and theoreti- 
cal work to do to improve the funding for and the science and practice of implementation. 

In the midst of all the complexities and challenges it is important to note that science-based programs and 
practices are being effectively used across the nation and internationally. While they are not yet fully scaled up, ac- 
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cess is increasing. Scholars, practice professionals, community members, policy makers, purveyors, and intermediaries 
need to j oin in the effort to understand and support best practices for intervention and for implementation. 
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but emphasize that such an approach is not sufficient for 
influencing the nature and quality of every-day community- 
based services, interventions, and systems. Anew, or at least 
substantially revised, discipline of "community science" is 
needed that will more effectively contribute to bridging 
research and scholarship on the one hand and community 
practice and policy on the other. 

Community science would have several character- 
istics: 

• A systematic process would plan, implement, and 
evaluate interventions (e.g., GTO), both their implementa- 
tion and outcomes, rather than simply replicating packaged 
interventions; 

• Researchers would collaborate with and support 
practitioners, patients, clients, and community leaders 
who would have the major responsibility for creating, 
implementing, and monitoring the interventions as well as 
being accountable for all the decisions they make; 

• An emphasis would be placed on having the techno- 
logical tools to disseminate science to practitioners and to 
provide comprehensive technical assistance to practitioners 
through an intermediate unit to help tailor knowledge and 
the change process to local circumstances. 

The focus of community science would be to add 
value (i.e., capacity), rather than simply research findings, 
to the community, and it would study the community change 
process to guide its improvement. For example, proponents 
identify certain issues that community science should ad- 
dress that are currently neglected: 

• Studies of policies and procedures that facilitate 
or hinder the adoption and implementation of effective 
interventions, the transition of demonstration programs to 
permanent and publicly available services, and the technol- 
ogy of effective information dissemination; 

• Studies of the effects of age, gender, ethnicity, and 
social- cultural factors that affect access to, use of, or ac- 
ceptance of preventive interventions; 

• Studies of the costs associated with the delivery of 
preventive interventions as well as methods of financing 
them; and 

• Studies of why model demonstration programs cre- 
ated by researchers tend to be less effective when they 



become publicly funded and implemented in the community 
by experienced service professionals in the context of real- 
world constraints. 

Aspen Institute's Roundtable on Community 
Change. The Aspen Institute's Roundtable (Auspos & Kubisch, 
2004) is another voice for comprehensive scholarly and 
community change. It seeks to develop a more coherent 
infrastructure for distilling and disseminating conclusions 
and lessons from recent and current initiatives, applying 
knowledge from related disciplines and fields, identifying 
new knowledge needs, and designing next-generation in- 
terventions and evaluations. It calls for the creation of a 
new learning cycle, that, in ideal form, moves away from 
the focus on single initiatives, integrates information from 
a full range of sources, organizes the information and as- 
sesses its value, takes the most promising elements and 
turns them into hypotheses, designs interventions that can 
test those hypotheses in a high-quality way, and evaluates 
the results. The Roundtable seeks to be a catalyst for change 
that recognizes and attempts to cope with the realities and 
requirements of both the nature of the research enterprise 
and the task of changing community systems. 

Conclusions 

After decades of largely being ignored and only 
rarely influential, applied scholars who work to improve 
human welfare directly are suddenly being looked to by 
service practitioners, funders, and policymakers who hope 
that research and scholarship can lead to more effective 
and financially efficient services and systems of support for 
high-risk citizens. This new attitude should be viewed by 
both groups as a rich opportunity, but it also constitutes a 
mandate to build a bridge over the chasm that has tradi- 
tionally separated these communities. To do so, all parties 
must realize that the bridge is complex and must not be 
oversimplified. Further, to cross the bridge will require new 
attitudes, methods, and research paradigms as well as a 
greater understanding of the complexity, cost, and length 
of time required to create the knowledge, services, and 
policies that will produce better outcomes and a better 
society. 
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Implications 

Acknowledging the limited research on many aspects 
of the process of implementing evidence-based programs 
and bringing them to scale in communities, the preceding 
discussion has several implications for consideration by 
researchers and practice and policy professionals. 

Researchers might consider the following: 

• Avoid emphasizing outcomes, especially in the first 
cohort of participants, until the process of implementa- 
tion is well established— documenting effective program 
procedures should be the first outcome. 

• Describe successful programs in sufficient detail so 
they can be replicated by others, including by paraprofes- 
sionals. 

• Attempt studies of the implementation of programs 
operated by staff similar to those who would operate the 
program if it were brought to scale, and describe variations 
in the program to fit local participants, budgets, and politi- 
cal circumstances. 

• When evaluating program implementation and 
outcome, consider a broader range of research designs, 
evidence of replication and generality, external as well 
as internal validity, various measures of effect size, and 
feasibility of replication and bringing the program to scale. 

• Collaborate with practice and policy professionals in 
formulating reviews of evidence that are not only scholarly 
but frame implications for practice and policy and fill-in the 
evidentiary blanks with best practices and best attainable 
solutions. 

• To the extent possible, attempt to study the es- 
sential characteristics of a program necessary to produce 
positive outcomes. 

• Begin to develop a science of community change that 
studies implementation by community professionals and 
how communities change their systems of human services. 

Practitioners and policymakers should consider the 
following integrated national and local approach: 

1. A coordinated set of interdisciplinary research, 
practice, and policy consensus groups, each focused on a 
maj or service/ policy need, should be created at the national 
level. They should be composed of scholars, practice pro- 
fessionals, policymakers, funders, and service clients with 
a strong, task-oriented, neutral leader who is sensitive to 
the values and unique contributions of each constituency. 
The charge for each group is to conduct a needs assess- 
ment (e.g., extent of the problem or need, prognosis 
and consequences of inaction, resources and personnel 
available relative to the need, and current approaches 
and effectiveness), a review of the evidence of promising 
approaches, and recommended guidelines for effective ser- 
vices or interventions and appropriate measures . The latter 
should be composed by the entire interdisciplinary group 
using a structured process consisting of the most promising 
elements of consensus groups, GTO and logic models, and 
Pathways, for example. The product should incorporate the 
research evidence, identify limits and gaps in the research 
literature, fill those gaps with best-evidence-available 



practices and consensus wisdom, identify characteristics of 
effective programs, and provide guidelines for one or more 
service options for potential implementation at federal, 
state, and/ or local levels. The specificity of the guidelines 
will vary with the research basis and the documented or 
estimated need for variations to fit different client groups 
and local social and political circumstances. The guidelines 
should also contain principles for implementation and pro- 
gram process and outcome evaluation that blend the need 
for common design features and measures with local varia- 
tions as well as a template for an application for funding 
to be made by communities and agencies to government 
or private funders that helps to structure locally proposed 
services to fit the guidelines. These consensus groups should 
be revisited periodically because social needs, research, 
and practice principles change over time. 

2. Government and private funders should use the 
guidelines and application template in combination with 
making technical assistance available to communities and 
agencies to conduct local planning and program develop- 
ment activities. The latter should involve the same inter- 
disciplinary mix of people and the same general logic model 
process as the consensus groups described above, except 
the guidelines would provide an evidence-based outline as 
the basis for possible services and activities to be adapted 
to local needs. Further, technical assistance should be made 
available if needed by independent and experienced "pur- 
veyors" to lead the local planning group, design the local 
evaluation, and meet the criteria of the funding application. 

3. Applications for funding could be judged in part on 
the extent to which the local proposals fit the guidelines 
on the one hand and propose reasonable and appropriate 
local variations on the other. 

4. Once funding is awarded, additional technical 
assistance, perhaps by having experienced purveyors col- 
laborate with local agencies, should be made available as 
needed to assist agencies and communities to implement 
their proposed project (e.g., see Fixsen et ah, 2005; McCall, 
Green, Strauss, &Groark, 1997; Groark & McCall, 2008) and 
especially to conduct the evaluation. 

5. Scholars and practice professionals, especially 
those engaged in the above processes, must vigorously en- 
courage the development of the research base for such an 
enterprise, including feeding the results of the evaluations 
of these programs back into the ongoing consensus groups, 
identifying and studying gaps in the service research, 
encouraging a broad range of methodologies, developing 
a science that studies implementation and practice per 
se, broadening the academic value system to more fully 
embrace quality work in this domain, and studying this 
process in the manner advocated by the community science 
proponents (e.g., Auspos & Kubish, 2004; Chinman et al., 
2005; Wandersman, 2003; Wandersman & Florin, 2003). 

Footnote 

'The author thanks Iisbeth B. Schorr for her substantial contributions 
to an early draft of the PM section of this paper; to Abraham Wanders- 
man and Anne Kubisch for their helpful comments on an earlier draft of 
this manuscript; and to IVfertha Zaslow for her patience, understanding, 
perspectives, and valuable suggestions. 
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